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EVALUATION  OF  A  PILOT  SOCIAL  WORK  OUTREACH  PROGRAM  USING  COMBAT 
AND  OPERATION  AL  STRESS  CONTROL  PRINCIPLES  IN  GARRISON 

FORT  SILL,  OKLAHOMA 
JUNE-SEPTEMBER  2009 

1.  PURPOSE.  The  U.S.  Army  Center  for  Health  Promotion  and  Preventive  Medicine 
(USACHPPM)  conducted  an  evaluation  of  a  pilot  social  work  outreach  program  (OP)  in  garrison 
at  Fort  Sill,  Oklahoma  using  both  quantitative  and  qualitative  data.  The  goal  of  this  evaluation 
was  threefold-.  (1)  to  describe  the  OP’s  activities  and  the  process  of  implementation,  (2)  to 
determine  the  OP’s  effect  on  Soldiers’  access  to  behavioral  health  (BH)  services,  and  (3)  to 
measure  the  OP’s  impact  on  Soldiers’  BH  outcomes  for  those  who  access  services. 

2.  CONCLUSIONS.  Overall,  the  OP  is  a  valued  and  well-received  BH  program  which  has  the 
capability  to  affect  a  positive  impact  on  Soldiers’  wellbeing.  Conclusions  specifically  relating  to 
OP  process,  impact,  and  outcome  measures  are  described  below. 

a.  The  primary  mission  of  tire  OP  is  to  pi'ovide  continual  BH  education  and  counseling 
awareness  services  to  Soldiers  at  Fort  Sill.  The  intent  is  to  bridge  the  gap  between  Soldiers  and 
BH  services.  The  OP’s  focus  on  proactive  activities,  including  trainings,  screenings,  and  sensing 
sessions,  increases  the  visibility  of  BH  in  Soldiers’  work  areas.  This  may  develop  trust  in  BH 
professionals  and  decrease  stigma. 

b.  Since  February  2007,  the  OP  has  made  contact  with  over  1 7,000  Soldiers,  Unit  Leaders 
and  Family  Readiness  Group  (FRG)  members  through  trainings,  formal  sensing  sessions,  and 
consultations.  Each  contact  is  an  opportunity  to  create  awareness  of  BH  programs  and  detect  and 
respond  to  BH  issues  before  they  affect  overall  readiness. 

c.  Soldiers,  Unit  Leaders,  and  FRG  members  who  have  had  contact  with  the  OP  report  high 
satisfaction  with  the  services  received.  The  program  is  particularly  appealing  because  of  its 
“boots  on  the  ground”  approach,  its  flexibility  in  scheduling  program  activities,  and  its  reputation 
for  consistent  follow  through. 

d.  Soldiers  and  Unit  Leaders  who  were  surveyed  by  the  OP  were  most  concerned  about 
being  treated  differently  by  their  leadership,  losing  confidence  from  members  of  their  unit,  and 
being  perceived  as  weak.  Among  those  surveyed  by  the  OP,  perceptions  of  barriers  to  BH  care 
were  generally  low. 
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e.  While  causality  cannot  be  established  at  this  time,  this  evaluation  suggests  a  positive 
impact  of  the  OP  on  Soldiers’  BH  utilization.  Stabilization  in  hospitalizations  and  an  increase  in 
ambulator)'  visits  for  mental  health  diagnoses  at  Reynolds  Army  Community  Hospital  during  OP 
implementation  may  be  associated  with  the  program’s  ability  to  intervene  early  and  encourage 
Soldiers’  use  of  BH  services  for  routine  issues  before  they  become  emergencies. 

f.  Soldiers  treated  through  the  OP  are  on  average  experiencing  a  mild  level  of  dysfunction 
similar  to  those  treated  through  Social  Work  Services  for  self-identified  relationship  issues.  This 
indicates  that  OP  staff  is  identifying  Soldiers  who  need  assistance  but  may  not  be  seeking  BH 
services  for  reasons  other  than  the  severity  of  their  problem. 

3.  RECOMMENDATIONS.  The  results  of  this  evaluation  support  the  continuation  and 
expansion  of  the  program  at  Fort  Sill  while  also  warranting  further  investigation  of  social  work 
OPs  in  garrison  and  their  effect  on  Soldiers’  general  wellbeing  and  mission  readiness. 
Recommendations  to  strengthen  the  body  of  evidence  for  the  effectiveness  of  OPs  in  garrison  are 
outlined  below. 

a.  Fort  Sill’s  Outreach  Program. 

(1)  Incorporate  a  long-term  evaluation  plan  into  standing  operating  procedures  (SOPs)  to 
include:  (1)  program  goals  and  objectives;  (2)  a  conceptual  framework;  (3)  process,  impact  and 
outcome  indicators;  (4)  a  data  collection  and  analysis  plan;  and  (5)  a  plan  for  dissemination  of 
future  evaluation  results. 

(2)  Begin  collection  of  OP-outcome  data  such  as  pre-  and  post-knowledge  of  topics 
covered  during  trainings  and  outcomes  for  Soldiers  who  received  outreach  services. 

(3)  Modify  current  and  develop  new  program  activities  to  address  Soldiers’  commonly 
cited  barriers  to  BH  care.  Evaluate  new  initiatives  for  effectiveness  with  regard  to  reducing 
stigma,  building  resiliency,  and  increasing  wellbeing. 

(4)  Increase  staffing  of  Licensed  Clinical  Social  Workers  and  Social  Work  Assistants  to 
strengthen  the  presence  of  the  OP  among  the  units  in  garrison  and  reduce  the  potential  for 
compassion  fatigue  among  staff. 

b.  Replication  and  Evaluation  Studies. 

(1)  Replicate  social  work  Ops  at  other  installations.  Fort  Sill’s  OP  was  developed  to 
meet  the  needs  of  that  particular  post.  In  addition,  the  success  of  the  program  appears  dependent 
on  the  quality  of  the  staffing.  It  is  important  to  determine  the  feasibility  of  implementing  this 
program  in  different  environments  with  different  populations  and  unit  structure.  Replication 
studies  would  be  especially  valuable  for  units  with  organic  BH  assets.  Coordination  between  OP 
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staff  and  organic  BH  assets  allow  for  improved  transition  of  the  OP  mission  from  garrison  to 
theater  and  return  to  garrison.  The  importance  of  this  continuity  cannot  be  understated. 

(2)  Include  an  evaluation  plan  in  SOP  for  any  replication  study.  At  a  minimum,  the 
evaluation  plan  should  include  a  protocol  for  collecting  data  on  program  activities  and  on 
expected  program  outcomes  before,  during,  and  following  full  program  implementation.  The 
ability  to  compare  expected  program  outcomes  before  and  after  implementation  strengthens 
evidence  to  gauge  program  effectiveness. 

(3)  Identify  a  lead  organization  and  point  of  contact  to  coordinate  evaluation  studies  of 
OP’s  at  other  installations.  This  will  ensure  similar  methods  of  data  collection  to  facilitate  the 
comparison  of  outcomes  across  programs  and  installations. 

(4)  Develop  and  disseminate  best  practices  for  OP  implementation  using  evidence  based 
upon  the  findings  from  multiple  evaluation  studies  and  sites. 
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1 .  REFERENCES.  See  Appendix  A  for  a  listing  of  references  used  in  this  report. 

2.  PURPOSE.  The  U.S.  Army  Center  for  Health  Promotion  and  Preventive  Medicine 
(USACHPPM)  conducted  an  evaluation  of  a  pilot  social  work  outreach  program  (OP)  in  garrison 
at  Fort  Sill,  Oklahoma  using  both  quantitative  and  qualitative  data.  The  goal  of  this  evaluation 
was  threefold:  (1)  to  describe  the  OP’s  activities  and  the  process  of  implementation,  (2)  to 
determine  the  OP’s  effect  on  Soldiers’  access  to  behavioral  health  (BH)  services,  and  (3)  to 
measure  the  OP’s  impact  on  Soldiers’  BH  outcomes  for  those  who  access  services. 

3.  AUTHORITY.  Website  request  from  Ms.  Cheryl  Kottke,  Outreach  Services  Coordinator, 
Fort  Sill’s  Outreach  Program,  5  May  2009,  subject:  Program  Evaluation  of  Fort  Sill’s  Outreach 
Program®. 

4.  BACKGROUND. 

a.  Project  Personnel.  Dr.  Jennifer  Piver-Renna,  a  public  health  researcher  with  the 
USACHPPM  Public  Health  Assessment  Program,  performed  an  evaluation  of  the  pilot  OP  at 
Fort  Sill,  Oklahoma  from  June  2009-September  2009. 

b.  Literature  Review.  A  review  of  the  published  literature  on  BH  OPs  indicates  the  value  of 
such  programs,  especially  following  traumatic  events®.  A  Cochrane  review  found  that 
participants  receiving  outreach  were  significantly  more  likely  to  stay  in  contact  with  BH  services 
and  were  significantly  less  likely  to  be  admitted  to  a  hospital  than  those  receiving  standard 
community  care®.  Implementation  of  a  behavioral  outreach  worker  program  for  adolescents  in 
need  of  BH  services  resulted  in  shorter  waiting  times  for  initial  BH  visits  and  increased  access  to 
mental  health  intervention  services®.  In  addition,  a  review  of  studies  on  BH  service  engagement 
concluded  that  the  model  of  service  delivery  was  more  important  than  patient  factors  when 
predicting  engagement  with  BH  services  and  that  outreach  models  including  flexible  hours,  short 
waiting  times,  and  frequent  contact  with  a  single  worker  can  maximize  engagement®. 
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c.  Outreach  Program  Description. 

(1)  Organization.  Fort  Sill’s  OP  is  assigned  to  the  Social  Work  Services  (SWS)  clinic 
within  Reynolds  Army  Community  Hospital  (RACH)  in  Oklahoma.  The  OP  was  formally 
initiated  in  October  2006  and  began  implementing  program  activities  in  February  2007. 
Appendix  B  shows  the  conceptual  framework  of  the  OP^f 

(2)  Mission.  The  mission  of  the  Soldier-centric  OP  is  to  plan,  conduct,  and  provide 
continual  education  and  counseling  awareness  services  in  support  of  Active  Duty  personnel  as 
well  as  National  Guard  and  Reserve  units  during  deployment  phases  while  assigned  to  Fort  Sill. 
It  conducts  this  mission  using  the  Combat  and  Operational  Stress  Control  (COSC)  principles  of 
prevention,  detection,  assessment,  normalization,  support,  and  referral1'7'1.  The  COSC 
interventions  are  commonly  implemented  in  deployed  environments;  however.  Fort  Sill’s  OP  is 
unique  in  its  application  of  these  principles  in  garrison. 

(3)  Staffing.  Table  1  shows  current  staff  positions  allocated  for  the  OP  and  their  primary 
responsibilities. 


Table  1.  Outreac 

i  Program  Staffing  Levels  and  Responsibilities 

Title 

Number 

Status 

Primary  Responsibilities 

Outreach  Program 
Coordinator 
(Licensed  Clinical 
Social  Worker 
(LCSW)) 

1 

Filled 

-  Coordinates  OP 

-  Performs  social  work  services 

-  Receives  guidance  and  direction  from  Chief  of  SWS 

-  Reviews  program  objectives  with  regard  to  adherence  to  Army¬ 
wide  goals  and  professional  social  work  principles 

Outreach  LCSW 

2 

Filled 

-  Assists  Program  Coordinator  with  OP  implementation 

-  Adheres  to  guidance  and  direction  from  Program  Coordinator 

-  Assists  in  implementing  program  objectives  and  Soldier 
Readiness  Processing  (SRP)  duties 

Outreach  Social 
Work  Assistant 
(SWA) 

2 

Filled 

-  Adheres  to  guidance  and  direction  from  Program  Coordinator 

-  Renders  services  to  Fort  Sill  community  covering  a  range  of 
social  service  functions 

-  Assesses,  researches,  and  assists  in  developing  programs  for 
military  and  families 

(4)  Funding.  In  September  2008,  OP  staff  was  converted  from  Other  Contingency 
Operations-funded  positions  (formally,  Global  War  on  Terrorism)  to  permanent  General  Service 
positions  supported  by  RACH. 

(5)  Utilization  Measures.  A  system  of  codes  to  accurately  document  OP  activities  in 
Armed  Forces  Health  Longitudinal  Technology  (AHLTA)  is  still  being  developed.  In  addition  to 
AHLTA,  OP  is  using  a  modified  version  of  the  Combat  and  Operational  Stress  Control 
Workload  and  Reporting  System  (COSC-WARS)  to  track  their  numbers  for  RACH 
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accountability.  This  system  was  developed  to  capture  COSC  interventions  in  theater  and  has 
been  adapted  to  reflect  the  program’s  use  in  garrison.  Relative  Value  Units  (RVU)  are  generated 
during  Soldier  screenings  and  assessments  most  often  at  SRP  events  and  during  short-term  brief 
supportive  counseling  sessions  with  Soldiers. 

d.  Outreach  Program  Services. 

(1)  Primary  Prevention.  The  OP  staff  delivers  trainings  for  Soldiers  in  group  settings  to 
raise  awareness  of  BH  issues  and  increase  Soldiers’  ability  to  identify  problematic  symptoms  in 
themselves  and  others.  Training  topics  include  COSC  reactions,  suicide  prevention,  anger 
management,  stress  management,  and  sleep  hygiene.  These  trainings  are  incorporated  into  the 
reintegration  cycle  (1  day  and  90/120  days  post-deployment)  but  can  also  be  delivered  upon 
request  of  the  Unit  Leader. 

(2)  Secondary  Prevention.  The  OP  offers  several  services  to  detect  existing  BH  issues 
among  Soldiers.  These  include — 

(a)  SRP  Screenings.  The  LCSWs  from  OP  attend  all  SRP  and  Reverse-SRP  (R-SRP) 
events  to  screen  and  assess  Soldiers  for  potential  BH  issues.  Soldiers  are  referred  to  the  LCSW 
at  the  SRP  site  based  on  their  responses  to  the  Post-Deployment  Health  Assessment  (PDHA)  or 
Post-Deployment  Health  Reassessmentment  (PDHA  or  PDHRA)  questionnaire.  The  LCSW 
normalizes  the  behavior  or  refers  the  Soldier  to  other  services  as  appropriate. 

(b)  Consultations.  The  OP  consults  with  Unit  Leaders  and  Chaplains  to  identify  BH 
needs  within  the  unit  and  to  discuss  courses  of  action  for  Soldiers  with  specific  issues. 

(c)  Formal  Sensing  Sessions.  The  OP  staff  administers  a  short  survey  to  Soldiers  to 
measure  the  Unit’s  overall  welfare  in  the  areas  of  morale,  unit  conflict,  sleep  quality,  personal 
conflict,  and  substance  use/abuse.  The  survey  can  usually  be  administered,  analyzed,  and  the 
results  reported  back  to  Unit  Leaders  within  a  week.  See  Appendix  C  for  a  copy  of  the  unit 
morale  survey. 

(d)  Informal  Sensing  Sessions.  An  OP  team  consisting  of  one  LCSW  and  one  SWA 
travels  to  various  work  areas  on  post  and  engages  Soldiers  in  informal  conversations  about  BH 
issues  and  other  relevant  topics.  These  sessions  provide  Soldiers  with  an  opportunity  to  become 
familiar  with  BH  services  and  build  trust  among  OP  staff,  thereby,  helping  to  reduce  stigma  for 
seeking  help.  During  these  sessions,  the  OP  staff  is  discreet,  yet  attentive,  to  any  request  for 
assistance.  The  OP  staff  seeks  permission  from  Unit  Leaders  before  speaking  with  Soldiers  and 
ensures  that  there  is  minimal  impact  on  the  Unit’s  operations  during  their  visit. 
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(3)  Tertiary  Prevention.  The  OP  staff  has  protocols  in  place  to  assist  Soldiers  who  need 
assistance  or  treatment  with  BH  issues.  These  include — 

(a)  Referrals.  Patient  referral  services  are  provided  during  walk-ins,  appointments,  or 
crisis  situations  and  include  referrals  back  to  OP  or  to  Command,  Primary  Care,  BH,  Emergency 
Department,  or  Chaplain. 

(b)  Short-term  Supportive  Counseling.  Soldiers  referred  back  to  OP  receive  short-term 
counseling  sessions  from  LCS  Ws.  Soldiers  who  need  extended  treatment  are  referred  to  other 
BH  clinics  as  appropriate. 

(c)  Traumatic  Event  Management  and  Crisis  Intervention.  The  OP  staff  collaborates 
with  other  appropriate  services  as  necessary  to  stabilize  Soldiers  in  crisis  and  debrief  following 
critical  events  (CEs). 

5.  METHODS. 

a.  Study  Design.  This  is  a  retrospective  evaluation  of  an  OP  program  that  was  implemented 
in  October  2006.  The  USACHPPM  used  a  mixed-methods  design  basing  findings  on  both 
quantitative  and  qualitative  data.  Each  data  source  is  described  below. 

b.  Data  Sources. 

(1)  Program  Activities  Data.  The  OP  staff  has  collected  data  on  program  activities  since 
October  2006.  Data  include  the  number  of  OP  activities  provided  per  month,  the  number  of 
Soldiers  reached  and  referred  byOP  activities,  and  Soldiers’  disposition  after  referral. 

(2)  Questionnaires. 

(a)  Satisfaction  Survey.  The  OP  staff  developed  an  8-item  survey  to  measure  program 
satisfaction  among  Soldiers,  Unit  Leaders,  and  the  Family  Readiness  Group  (FRG).  Surveys 
were  distributed  to  attendees  after  most  program  activities  from  January  through  September 
2009.  Respondents  rated  their  experience  with  OP  staff  on  a  5-point  Likert  scale  and  were  also 
able  to  write  in  suggestions  for  program  improvement.  See  Appendix  D  for  the  Soldiers’ 
satisfaction  survey. 

(b)  Barriers  to  Care/Stigma  Survey.  In  September  2009,  the  OP  staff  attached  a  13-item 
survey  to  the  back  of  the  satisfaction  survey  measuring  barriers  (including  stigma)  to  accessing 
BH  services.  The  USACHPPM  adapted  the  barrier/stigma  survey  from  the  survey  used  during 
previous  Mental  Health  Advisory  Team  assignments.  The  OP  administered  the  survey  to 
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Soldiers  and  Unit  Leaders  after  most  program  activities.  See  Appendix  E  for  the  barriers  to 
care/stigma  survey. 

(c)  BH  Outcomes  Data.  The  Outcome  Questionnaire  (OQ®-45)  is  a  45-item  self-report 
questionnaire  that  measures  Soldiers’  functional  level  on  three  dimensions:  symptom  distress, 
interpersonal  functioning,  and  social  role(8l  It  can  be  used  to  track  the  progress  of  patients 
during  therapy  and  is  administered  at  the  beginning  of  each  counseling  session  in  all  Department 
of  Behavioral  Health  (DBH)  clinics  at  RACH  including  OP.  The  USACHPPM  compared  initial 
OQ®-45  scores  for  Soldiers  seeking  counseling  through  SWS  and  through  OP  since 
incorporation  of  the  OQ®-45  into  clinical  practice.  Effective  provision  of  appropriate  referrals 
for  care  by  the  OP  staff  would  be  evidenced  by  a  decrease  in  the  OP’s  initial  OQ®-45  scores  over 
time.  (The  OQ®-45  is  a  registered  trademark  of  OQ  Measures,  LLC). 

c.  Key  Informant  Interviews.  The  USACHPPM  conducted  five  interviews  during  August 
2009  with  current  or  former  staff  in  DBH  at  RACH  who  had  extensive  knowledge  of  OP.  The 
OP  staff  provided  contact  information  for  initial  informants.  The  USACHPPM  sought  contact 
information  for  additional  informants  from  each  informant  after  the  interview.  See  Appendix  F 
for  the  key  informant  interview  question  protocol. 

d.  Epidemiological  Behavioral  Health  Data. 

(1)  Through  the  Defense  Medical  Epidemiology  Database  (DMED)  of  the  Armed  Forces 
Health  Surveillance  Center  (AFHSC),  USACHPPM  obtained  data  on  hospitalizations  and 
ambulatory  visits  for  Soldiers  seen  at  RACH  with  an  International  Classification  of  Diseases,  9th 
Revision  (ICD-9)  primary  diagnosis  of  mental  disorder  from  1999  to  2008.  Effective  facilitation 
of  early  intervention  would  be  evidenced  by  a  decrease  in  hospitalizations. 

(2)  Through  the  TRICARE  Operations  Center  (TOC),  USACHPPM  obtained  data  on 
number  of  appointments,  appointment  status,  and  appointment  type  by  clinic  for  BH  services  at 
RACH  from  October  2006  July  2009.  Effective  facilitation  of  early  intervention  would  be 
evidenced  by  a  decrease  in  acute  BH  visits  and  an  increase  in  routine  BH  appointments. 

e.  Data  Limitations.  The  USACHPPM  conducted  this  evaluation  retrospectively  nearly  3 
years  after  program  inception.  As  such,  the  results  of  this  evaluation  are  largely  based  on  self- 
report  surveys,  process  data  available  from  program  staff,  or  health  outcome  data  from  large 
military  databases.  The  ability  to  make  meaningful  comparisons  to  outcomes  before  OP  was 
implemented  or  to  outcomes  from  similar  programs  is  limited.  These  limitations  temper  the 
strength  of  the  conclusions  that  can  be  drawn  from  the  data  with  regard  to  program  effectiveness 
and  impact. 
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f,  Data  Analysis.  The  Statistical  Package  for  the  Social  Sciences  (SPSS®),  Version  16.0, 
was  used  for  statistical  analysis.  Descriptive  statistics  (i.e.,  frequencies,  distributions,  and 
means)  were  calculated  for  questions  on  the  satisfaction  survey  and  barriers/stigma  survey.  T- 
tests  were  used  to  compare  initial  mean  OQ®-45  scores  between  OP  and  SWS.  (SPSS®  is  a 
registered  trademark  of  SPSS  Corporation). 

6.  FINDINGS. 

a.  Program  Activity  Data. 

(1)  The  complete  data  set  showing  the  number  of  OP  activities  per  month,  the  number  of 
contacts  reached  (i.e.,  Soldiers,  Leaders,  and  FRG  members)  for  each  activity  per  month,  and  the 
average  number  of  contacts  reached  for  each  activity  since  program  activities  began  in  February 
2007  is  available  in  Appendix  G. 

(2)  Figure  1  shows  the  average  number  of  contacts  for  each  formal  sensing  session  per 
month.  From  February  2007  to  July  2009,  an  average  of  63.28  contacts  completed  the  unit 
morale  survey  per  month. 


Figure  1.  Average  Number  of  Contacts  Per  Morale  Survey  Event 

(3)  Figure  2  shows  the  number  of  Unit  Leader  and  Soldier  consultations  per  month. 
From  February  2007  to  July  2009,  OP  staff  conducted  an  average  of  3.53  Unit  Leader 
consultations  and  14.93  Soldier  consultations  each  month. 
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Figure  2.  Unit  Leader  and  Soldier  Consultations  Per  Month 

(4)  Figure  3  shows  Soldiers’  dispositions  after  consultation  with  OP  staff.  Sixty-three 
percent  of  Soldiers  (n  ~288)  returned  to  duty  with  no  restrictions. 


Emergency  Room 


Figure  3.  Soldier  Disposition  after  OP  Consultation  (n  =  468) 
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(5)  Figure  4  shows  the  number  and  type  of  Soldier  referrals  given  by  OP  staff.  Most 
Soldiers  were  referred  to  either  a  primary  care  manager  (n  =  148,  36.10%)  or  to  Social  Work 
Services  (n  =  147,  35.85%). 
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Figure  4.  Soldier  Referrals  from  OP  (n  =  41 0) 


(6)  Figure  5  shows  the  average  number  of  contacts  attending  each  preventive  educational 
training  session  per  month.  From  February  2007  to  July  2009,  an  average  of  43.71  contacts 
attended  each  training  session. 
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Figure  5.  Average  Number  of  Contacts  Per  Training 

(7)  Table  2  shows  the  number  of  Soldiers  screened,  assessed,  and  briefed  by  OP  staff  at 
SRP  events  pre-  and  post-deployment  each  fiscal  year. 


Table  2.  Number  of  Soldiers  Assessed  and  Briefed  at  SRP  and  R-SRP  Events 


Date 

Pre-Deployment 

SRP 

Post-Deployment 

R-SRP 

FY  07:  October  2006  to  September  2007 

476 

1009 

FY  08:  October  2007  to  September  2008 

4489 

630 

Partial  FY  09:  October  2008  to  July  2009 

6600 

1235 

Total 

11565 

2874 

b.  Satisfaction  Survey. 

(1)  The  OP  staff  administered  satisfaction  surveys  to  185  contacts.  Of  those,  80  percent 
(n  =  147)  provided  information  about  their  experiences  with  OP.  Seventy-one  percent  (n  =  105) 
of  respondents  were  Soldiers,  19  percent  (n  =  28)  were  Unit  Leaders,  and  10  percent  (n=  14) 
were  FRG  members. 

(2)  Table  3  shows  the  mean  response  for  each  question  on  the  satisfaction  survey.  The 
response  set  for  each  question  ranged  from  1  (worst  possible)  to  5  (best  possible).  There  were  no 
significant  differences  in  self-reported  program  satisfaction  among  groups  (Soldiers,  Unit 
Leaders,  and  FRG). 
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Table  3.  Mean  Response  to  Satisfaction  Survey  Items  (n  =147) 


Question 

Mean  Response 

How  would  you  rate  your  overall  experience  with  FSOP? 

4.52 

Please  rate  the  FSOP  staffs  efforts  to  treat  you  with  courtesy  and  respect. 

4.31 

How  helpful  was  the  FSOP  team  or  team  member  you  encountered? 

4.69 

How  well  did  the  FSOP  team  or  team  member  listen  to  you? 

4.53 

How  well  did  the  FSOP  team  or  team  member  address  your  concerns? 

4.46 

How  well  did  the  FSOP  team  or  team  member  meet  the  needs  of  your  unit? 

4.48 

(3)  The  satisfaction  survey  also  asked  respondents  to  list  ways  in  which  the  program 
could  be  improved.  Table  4  shows  the  major  themes  of  the  comments  and  examples  in  each 
category. 


Table  4.  Comments  from  Satisfaction  Survey  (n  =  81) 


Theme 

Percent  of 
Responses 

Examples 

Praise 

51 

“Very  helpful,  friendly,  and  wanting  to  make  things  better  for  the 

Soldier.” 

“We  intend  to  fully  utilize  the  many  services  and  training  support  FSOP 
offers!” 

“Very  happy  to  have  this  available.” 

“Kept  Soldiers  engaged  and  it  was  very  interesting.” 

Additional  information 

23 

“More  information  on  stress  relief.” 

“More  scenarios,  maybe  role  play.” 

“Make  the  curriculum  more  scenario-driven.” 

“Address  timeline  of  symptoms  and  improvements.” 

Additional 

activities/resources 

19 

“More  counselors  would  better  help  us  serve  our  Soldiers.” 

“Provide  more  one-on-one  courses.” 

“Schedule  more  classes  at  different  times  so  Soldiers  out  training  can 
attend.” 

“Have  this  service  briefed  before  and  after  deployment.” 

Awareness 

7 

“Advertise  services  offered  through  flyers.” 

“Put  the  program  out  there  more  through  brigade  and  battalion 
commanders.” 

“Better  public  awareness.” 

“Didn’t  know  Fort  Sill  had  program.” 
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c.  Barrier  to  Care/Stigma  Survey. 

(1)  The  OP  staff  administered  a  survey  to  measure  banders  to  care,  including  stigma,  to 
61  contacts  during  September  2009.  Of  those,  74  percent  («  =  45)  were  Soldiers  and  26  percent 
(n=  16)  were  Unit  Leaders. 

(2)  Table  5  shows  the  mean  response  for  the  each  statement  on  the  survey  ordered  from 
highest  rated  to  lowest  rated  barrier.  The  response  set  for  each  question  ranged  from  1  (strongly 
disagree)  to  5  (strongly  agree).  There  were  no  significant  differences  in  self-reported  barriers  to 
care  between  Soldiers  and  Unit  Leaders. 


Table  5.  Mean  Response  to  Barriers  to  Care  Survey  Items  (n  =61) 


Question 

Mean  Response 

Leadership  might  treat  me  differently. 

2.59 

Members  of  my  unit  might  have  less  confidence  in  me. 

2.51 

I  would  be  seen  as  weak. 

2.48 

It  would  harm  my  career. 

2.28 

I  don’t  trust  mental  health  professionals. 

2.23 

It  would  be  too  embarrassing. 

2.16 

My  leaders  would  blame  me  for  the  problem. 

There  would  be  difficulty  getting  time  off  work  for  treatment. 

2.08 

It  is  difficult  to  schedule  an  appointment. 

2.00 

Mental  health  care  doesn’t  work. 

Mental  health  care  costs  too  much  money. 

1.75 

I  don’t  know  where  to  get  help. 

1.44 

I  don’t  have  adequate  transportation. 

1.30 

d.  Hospitalizations.  Figure  6  shows  the  rate  of  hospitalization  at  RACH  for  Soldiers  with  a 
primary  diagnosis  of  mental  disorder  per  ICD-9  coding  criteria  from  1999  to  2008.  Overall,  the 
rate  of  hospitalization  has  increased  from  4.08  hospitalizations  per  1,000  Soldiers  in  1999  to 
17.83  hospitalizations  per  1,000  Soldiers  in  2008.  The  rate  of  hospitalizations  stabilized  during 
the  two  years  of  OP  implementation  with  17.76  hospitalizations  per  1,000  Soldiers  in  2007  and 
17.83  hospitalizations  per  1,000  Soldiers  in  2008. 
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Figure  6.  Rate  of  Soldier  Hospitalizations  with  an  ICD-9  Primary  Diagnosis 

of  Mental  Disorder 


e.  Ambulatory  Care.  Figure  7  shows  the  rate  of  ambulatory  visits  at  RACH  for  Soldiers 
with  a  primary  diagnosis  of  mental  disorder  per  ICD-9  coding  criteria  from  1999  to  2008. 
Overall,  the  rate  of  ambulatory  visits  has  increased  from  389.28  visits  per  1,000  Soldiers  in  1999 
to  1230.28  visits  per  1,000  Soldiers  in  2008.  The  rate  of  ambulatory  visits  increased  during  the 
two  years  of  OP  implementation  with  823.01  visits  per  1,000  Soldiers  in  2007  to  1230.28  visits 
per  1,000  Soldiers  in  2008. 
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Figure  7.  Rate  of  Soldier  Ambulatory  Visits  with  an  ICD-9  Primary  Diagnosis 

of  Mental  Disorder 


f.  Behavioral  Health  Visits. 

(1)  Figure  8  shows  the  number  of  Soldiers’  acute  BH  visits  at  RACH  from  4th  quarter 
2006  to  2nd  quarter  2009.  The  data  are  aggregated  into  quarters  due  to  small  numbers.  Overall, 
the  number  of  Soldiers’  acute  BH  visits  decreased  from  48  visits  in  4th  Quarter  2006  to  5  in  2nd 
quarter  2009. 


60 


2006  2007  2007  2007  2007  2007  2008  2008  2008  2009  2009 

*Note:  Q2  of 2007  only  includes  data  from  the  months  of  April  and  June  due  to  missing  data  for  May. 

Figure  8.  Number  of  Acute  Behavioral  Health  Visits 
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(2)  Figure  9  shows  the  number  of  routine,  specialty  and  wellness  (non-acute)  BH  visits  at 
RACH  from  4th  quarter  2006  to  2nd  quarter  2009.  The  data  are  aggregated  into  quarters  due  to 
small  numbers.  Overall,  the  number  of  Soldiers’  non-acute  BH  visits  also  decreased  from  5683 
visits  in  4th  quarter  2006  to  2341  visits  in  2nd  quarter  2009. 


6000 


Figure  9.  Number  of  Non-Acute  Behavioral  Health  Visits 


g.  Behavioral  Health  Outcomes.  The  OQ-45  scores  for  individual  Soldiers  were  not 
available  at  the  time  of  this  report.  Therefore,  trends  in  scores  over  time  controlling  for  time  in 
treatment  could  not  be  determined.  However,  OQ-45  scores  at  initial  visits  with  SWS  and  OP 
were  available.  Figure  10  shows  the  average  OQ-45  scores  for  Soldiers  during  their  first  visit 
with  OP  or  SWS.  The  scoring  manual  states  that  scores  above  63  indicate  dysfunction®.  On 
average,  OQ-45  scores  for  Soldiers  seen  at  OP  and  SWS  were  not  significantly  different  (i( 8)  = 
1.80,  n.s.). 
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Figure  10.  OQ-45  Scores  for  OP  and  SWS 

h.  Program  Implementation  Experiences.  During  key  informant  interviews,  RACH  OP  and 
DBH  staff  identified  several  barriers,  facilitators,  and  lessons  learned  of  program 
implementation.  Table  6  identifies  major  themes  in  each  category  and  supporting  quotes. 

Table  6.  Lessons  Learned  from  OP  Implementation 

Theme  Quotes 

Barriers 

“Trying  to  merge  OP  with  the  military  treatment  facility  (MTF)  accountability  system  is  like 
Accountability  putting  a  round  object  in  a  square  hole.” 

system  “The  bean  counters  were  having  a  tough  time  wrapping  their  heads  around  the  idea  that  they 

have  clinical  social  workers  who  weren’t  being  clinical  per  se.” 

“When  you  throw  the  MTF  in  and  the  responsibilities  it  has  to  accountability,  I  think  then  that 
adjusts  the  people  in  the  leadership  positions’  viewpoints  because  they  become  uncomfortable 
with  [the  unstructured  pieces  of  the  program].” 

“Getting  out  in  front  of  things  instead  of  always  being  reactive.” 

“Civilians  in  my  experience  are  very  uncomfortable  at  first  with  the  whole  nuanced  thing  of 
walking  down  [into  the  units].” 

“There  is  pressure  to  become  more  clinical.” 

Facilitators 


Executing 

mission 
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Table  6.  Lessons  Learned  from  OP  Implementation  (continued) 


Theme 

Quotes 

Program 

champions 

“Not  only  did  [the  program  manager]  kick  down  the  door  but  what  that  created  was  static — a  lot 
of  static — among  the  brigade  commander  to  motivate  the  MTF  to  continue  the  program.” 

“They  said,  ‘Do  you  want  to  try  and  make  this  thing  work?’  and  [Chief,  SWS]  said  ‘I  have  to 
have  it  my  way  and  y’all  need  to  leave  me  alone  and  let  me  just  do  it’.” 

“The  program  manager  got  the  vision  early  and  is  self-motivated.” 

Perceived  benefit 

“The  program  increases  workload  in  some  ways  and  decreases  workload  in  other  ways.  Overall, 
however,  it  maximizes  care  for  behavioral  health  issues.” 

“When  we  do  referrals  to  behavioral  health  they  are  more  appropriate,  more  legitimate.” 

“It  takes  behavioral  health  out  of  the  ivory  tower.” 

“You  are  down  at  the  smallest  group  level  facilitating  improvement  and  communications  within 
behavioral  health  services.” 

Flexibility 

“If  you  are  going  to  go  out  there  and  work  with  the  Soldiers  and  the  command  team  you  have  to 
be  there  on  their  time,  not  on  your  time.” 

“It’s  flexible.  It  works  with  the  unit.  It’s  not  rigid  like  a  clinic — you  open,  you  close.” 

“You  are  going  to  have  to  have  some  people  who  are  flexible,  willing  to  sometimes  go  above 
and  beyond,  and  who  truly  love  doing  community  social  work.” 

Lessons  learned 

Follow  through 

“If  you  promise  something,  you  better  deliver  it.” 

“We  are  showing  them  that  our  word  is  our  bond.” 

“If  we’ve  kept  through  to  our  word  then  we  get  the  respect  of  the  command  team  and  they  will 
use  us.” 

“The  minute  we  start  not  following  through  and  take  shortcuts  we  are  going  to  end  up  flat  on 
our  face.” 

Be  effective 

“Start  small,  earn  respect,  and  be  effective.” 

“Everybody  is  getting  the  word  out  that  if  they  want  something  done,  you  call  and  ask  us  to 
help.” 

“There  is  a  trust  with  us.  They  realize  that  what  they  said  meant  something  and  we  did  what 
they  asked  and  so  they  come  back  [from  deployment]  and  ask  specifically  for  us.” 

Find  the  right 
staff 

“Staff  drives  the  direction  of  the  program  and  can  make  it  or  break  it.” 

“It  is  a  special  kind  of  person  to  stay  with  this  and  not  get  discouraged.” 

“It  is  a  necessity  to  find  personnel  with  very  good  clinical  boundaries.” 

“The  best  possible  combination  would  be  a  social  work  officer  with  COSC  experience  and  their 
civilian  counterpart  who  has  a  community  mindset.” 

7.  CONCLUSIONS.  Overall,  the  OP  is  a  valued  and  well-received  BH  program  which  has  the 
capability  to  affect  a  positive  impact  on  Soldiers’  wellbeing.  Conclusions  specifically  relating  to 
OP  process,  impact,  and  outcome  measures  are  described  below. 

a.  The  primary  mission  of  the  OP  is  to  provide  continual  BH  education  and  counseling 
awareness  services  to  Soldiers  at  Fort  Sill.  The  intent  is  to  bridge  the  gap  between  Soldiers  and 
BH  services.  The  OP’s  focus  on  proactive  activities,  including  trainings,  screenings,  and  sensing 
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sessions,  increases  the  visibility  of  BH  in  Soldiers’  work  areas.  This  increased  presence  was 
addressed  directly  by  Lieutenant  General  Schoomaker  as  one  action  Leaders  could  take  to 
address  barriers  to  wellness,  especially  stigma,  that  may  impact  mission  readiness{9). 

b.  Since  February  2007,  the  OP  has  made  contact  with  over  1 7,000  Soldiers,  Unit  Leaders 
and  FRG  members  through  trainings,  formal  sensing  sessions,  and  consultations.  Each  contact  is 
an  opportunity  to  create  awareness  of  BH  programs  and  detect  and  respond  to  BH  issues  before 
they  affect  overall  readiness 

c.  Soldiers,  Unit  Leaders,  and  FRG  members  who  have  had  contact  with  the  OP  report  high 
satisfaction  with  the  services  received.  The  program  is  particularly  appealing  because  of  its 
“boots  on  the  ground”  approach,  its  flexibility  in  scheduling  program  activities,  and  its  reputation 
for  consistent  follow  through. 

d.  Soldiers  and  Unit  Leaders  who  were  surveyed  by  the  OP  were  most  concerned  about 
being  treated  differently  by  their  Leadership,  losing  confidence  from  members  of  their  unit,  and 
being  perceived  as  weak.  However,  the  perception  of  barriers  among  Soldiers  surveyed  by  OP 
were  much  lower  than  perceptions  measured  in  other  published  literature  of  previously  deployed 
infantry  units(!0’ 1 and  spouses  of  deployed  service  members^12-*. 

e.  While  causality  cannot  be  established  at  this  time,  this  evaluation  suggests  a  positive 
impact  of  the  OP  on  Soldiers’  BH  utilization.  Stabilization  in  hospitalizations  and  an  increase  in 
ambulatory  visits  for  mental  health  diagnoses  at  RACH  during  OP  implementation  may  be 
associated  with  the  program’s  ability  to  intervene  early  and  encourage  Soldiers’  use  of  BH 
services  for  routine  issues  before  they  become  emergencies.  During  the  same  time  frame, 
hospitalizations  for  all  ICD-9  diagnoses  showed  a  similar  pattern  of  stabilization  while 
ambulatory  rates  for  all  ICD-9  diagnoses  decreased.  However,  the  data  do  not  account  for  other 
unknown  or  known  factors,  such  as  operational  tempo,  service  availability,  or  other  health  care 
programs  that  may  also  affect  these  rates. 

f.  Although  acute  BH  visits  decreased  as  expected,  non-acute  (i.e.,  routine,  specialty,  and 
wellness)  BH  visits  also  decreased  over  time  contrary  to  the  pattern  expected.  For  both  acute 
and  non-acute  visits,  rates  of  Soldiers’  visits  were  not  available  and  counts  were  presented 
instead.  The  data,  therefore,  are  more  likely  to  be  influenced  by  deployment  cycle  and  may  not 
be  representative  of  actual  BH  service  utilization. 

g.  Although  trends  in  OQ-45  scores  over  time  were  not  available,  the  data  did  show  that 
Soldiers  treated  through  OP  are  on  average  experiencing  a  mild  level  of  dysfunction  similar  to 
those  treated  through  SWS  for  self-identified  relationship  issues.  This  indicates  that  OP  staff  are 
identifying  Soldiers  who  need  assistance  but  may  not  be  seeking  BH  care  for  reasons  other  than 
severity  of  their  problem. 
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8.  RECOMMENDATIONS.  The  results  of  this  evaluation  support  the  continuation  and 
expansion  of  the  program  at  Fort  Sill  while  also  warranting  further  investigation  of  social  work 
OPs  in  garrison  and  their  effect  on  Soldiers’  general  wellbeing  and  mission  readiness. 
Recommendations  to  strengthen  the  body  of  evidence  for  the  effectiveness  of  OPs  in  garrison  are 
outlined  below. 

a.  Fort  Sill’s  Outreach  Program. 

(1)  Incorporate  a  long-term  evaluation  plan  into  SOPs  to  include:  (1)  program  goals  and 
objectives;  (2)  a  conceptual  framework;  (3)  process,  impact  and  outcome  indicators;  (4)  a  data 
collection  and  analysis  plan;  and  (5)  a  plan  for  dissemination  of  future  evaluation  results. 

(2)  Begin  collection  of  OP  outcome  data,  such  as  pre-  and  post-knowledge  of  topics 
covered  during  trainings  and  outcomes  for  Soldiers  who  received  outreach  services.  See 
Appendix  H  for  a  list  of  suggested  metrics  and  measures. 

(3)  Modify  current  and  develop  new  program  activities  to  address  Soldiers’  commonly 
cited  barriers  to  BH  care.  Evaluate  new  initiatives  for  effectiveness  with  regard  to  reducing 
stigma,  building  resiliency,  and  increasing  wellbeing. 

(4)  Increase  staffing  LCS  W  and  SWAs  to  strengthen  the  presence  of  the  OP  among  the 
units  in  garrison  and  reduce  the  potential  for  compassion  fatigue  among  staff. 

b.  Replication  and  Evaluation  Studies. 

(1)  Replicate  social  work  OPs  at  other  installations.  Fort  Sill’s  OP  was  developed  to 
meet  the  needs  of  that  particular  post.  In  addition,  the  success  of  the  program  appears  dependent 
on  the  quality  of  the  staffing.  It  is  important  to  determine  the  feasibility  of  implementing  this 
program  in  different  environments  with  different  populations  and  unit  structure.  Replication 
studies  would  be  especially  valuable  for  units  with  organic  BH  assets.  Coordination  between  OP 
staff  and  organic  BH  assets  allow  for  improved  transition  of  the  OP  mission  from  garrison  to 
theater  and  return  to  garrison.  The  importance  of  this  continuity  cannot  be  understated. 

(2)  Include  an  evaluation  plan  in  SOP  for  any  replication  study.  At  a  minimum,  the 
evaluation  plan  should  include  a  protocol  for  collecting  data  on  program  activities  and  on 
expected  program  outcomes  before,  during,  and  following  full  program  implementation.  The 
ability  to  compare  expected  program  outcomes  before  and  after  implementation  strengthens 
evidence  to  gauge  program  effectiveness. 
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(3)  Identify  a  lead  organization  and  point  of  contact  to  coordinate  evaluation  studies  of 
OP’s  at  other  installations.  This  will  ensure  similar  methods  of  data  collection  to  facilitate  the 
comparison  of  outcomes  across  programs  and  installations. 

(4)  Develop  and  disseminate  best  practices  for  OP  implementation  using  evidence  based 
upon  the  findings  from  multiple  evaluation  studies  and  sites. 

9.  POINT  OF  CONTACT.  Dr.  Jennifer  Piver-Renna,  the  principal  investigator,  is  the  point  of 
contact  for  this  project.  She  may  be  reached  at  410-436-9283  (commercial)  or  584-9283  (DSN) 
or  by  email  atjennifer.piverrenna@us.army.mil. 


JENNIFER  M.  PIVER-RENNA,  PHD 

Public  Health  Researcher 

Public  Health  Assessment  Program 


Reviewed  by: 


STEVEN  H.  BULLOCK,  DPT 

Program  Manager 

Public  Health  Assessment  Program 
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APPENDIX  B 

CONCEPTUAL  FRAMEWORK  OF  FORT  SILL’S  OUTREACH  PROGRAM 
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APPENDIX  D 

SOLDIER  SATISFACTION  SURVEY 

The  purpose  of  this  survey  is  to  collect  information  about  your  experience  with  the  Fort  Sill 
Outreach  Program  (FSOP).  Your  responses  will  be  confidential.  The  information  you  provide 
will  be  used  to  improve  the  care  and  other  services  we  provide.  Thank  you  for  your  assistance. 

Date: _ 

1)  Circle  your  Unit:  75th  214th  434th  428th  479th  USMC  FA  Det. 

Please  answer  the  following  questions  by  circling  the  number  that  best  corresponds  to  your 
assessment  of  the  FSOP,  where  1  is  the  worst  possible  and  5  is  the  best  possible. 

2)  How  would  you  rate  your  overall  experience  with  the  FSOP? 

1  (worst  possible)  2  3  4  5  (best  possible) 

3)  Please  rate  the  FSOP  staffs  efforts  to  treat  you  with  courtesy  and  respect. 

1  (worst  possible)  2  3  4  5  (best  possible) 

4)  How  helpful  was  the  FSOP  team  or  the  team  member  you  encountered? 

1  (worst  possible)  2  3  4  5  (best  possible) 

5)  How  well  did  the  FSOP  team  or  team  member  listen  to  you? 

1  (worst  possible)  2  3  4  5  (best  possible) 

6)  How  well  did  the  FSOP  team  or  team  member  address  your  concerns? 

1  (worst  possible)  2  3  4  5  (best  possible) 

7)  How  well  did  the  FSOP  team  or  team  member  meet  the  needs  of  your  unit? 

1  (worst  possible)  2  3  4  5  (best  possible) 

8)  How  could  we  improve  our  services? 
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APPENDIX  E 

BARRIER  TO  CARE/STIGMA  SURVEY 

Please  rate  each  of  the  following  factors  that  might  affect  your  decision  to  receive  behavioral 
health  counseling  or  services  if  you  ever  had  a  problem  on  a  scale  from  STRONGLY 
DISAGREE  to  STRONGLY  AGREE. 


Strongly 

Disagree 

Disagree 

Neither 
Agree  nor 
Disagree 

Agree 

Strongly 

Agree 

I  don’t  trust  mental  health 
professionals. 

1 

2 

3 

4 

5 

I  don’t  know  where  to  get  help. 

1 

2 

3 

4 

5 

I  don’t  have  adequate  transportation. 

1 

2 

3 

4 

5 

It  is  difficult  to  schedule  an 
appointment. 

1 

2 

3 

4 

5 

There  would  be  difficulty  getting 
time  off  work  for  treatment. 

1 

2 

3 

4 

5 

Mental  health  care  costs  too  much 
money. 

1 

2 

3 

4 

5 

It  would  be  too  embarrassing. 

1 

2 

3 

4 

5 

It  would  harm  my  career. 

1 

2 

3 

4 

5 

Members  of  my  unit  might  have  less 
confidence  in  me. 

1 

2 

3 

4 

5 

My  unit  leadership  might  treat  me 
differently. 

1 

2 

3 

m 

5 

My  leaders  would  blame  me  for  the 
problem. 

1 

2 

3 

n 

5 

I  would  be  seen  as  weak. 

2 

3 

4 

5 

Mental  health  care  doesn’t  work. 

1 

2 

3 

4 

5 
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APPENDIX  F 

KEY  INFORMANT  INTERVIEW  QUESTION  PROTOCOL 


1 .  What  is  your  role  in  the  Fort  Sill  Outreach  Program? 

2.  Could  you  describe  the  Fort  Sill  Outreach  Program? 

□ 

Organization  within  RACH? 

□ 

Program  activities? 

□ 

Staffing? 

□ 

Counseling/ consultations? 

□ 

Referrals? 

□ 

Follow-up  procedures  with  patient? 

□ 

Funding/financial  support? 

3.  How  long  have  you  been  involved  in  the  program? 

4.  What  were  some  of  the  driving  factors  that  led  the  implementation  of  this  program? 

5 .  How  has  the  program  changed  over  time  since  you  became  involved? 

6.  What  role,  if  any,  did  your  experience  as/with  Soldiers  play  in  your  awareness  of  or  decision 
to  implement  the  outreach  program? 

7.  What  do  you  consider  some  of  the  benefits  of  the  outreach  program? 

□  To  RACH? 

1  |  To  Soldiers? 

□  To  Big  Army? 

8.  What  was  the  goal  of  this  program?  Has  that  goal  been  reached? 

9.  What  were  some  concerns  or  problems  that  the  hospital  anticipated  when  implementing  the 
outreach  program? 

I  I  Effectiveness  of  intervention  in  garrison  conditions? 

[  Ease  of  use? 

I  i  Acceptability  by  staff? 

| _ j  Acceptability  by  Soldiers? 

I  I  Adequacy  of  existing  infrastructure  and  human  resources? 
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10.  What  factors  have  supported  the  continued  implementation  of  the  outreach  program? 

□  Influence  of  a  program  champion? 

j_j  Sufficient  financial  resources? 

L  J  Well  planned  implementation  approach? 

!_j  Involvement  of  end  users  in  implementation  planning  and  execution? 

I  I  Fit  with  other  organizational  goals  and  programs? 

1 1 .  What  challenges  arose  in  implementing  the  outreach  program? 

!  |  Poor  acceptability  by  staff? 

I  I  Poor  acceptability  by  Soldiers? 

M  Insufficient  financial  resources? 

CH  Inadequate  infrastructure? 

I  I  Incompatibility  with  other  organizational  goals  and  programs? 

12.  What  consequences,  good  or  bad,  has  RACH  experienced  from  implementing  this  outreach 
program? 

[ _ !  Consequences  for  Soldiers? 

I  I  Consequences  for  staff? 

I  I  Consequences  for  hospital  as  a  whole? 

13.  Did  you  learn  any  important  lessons  from  the  implementation  of  the  outreach  program? 

14.  Would  you  recommend  this  program  at  other  installations?  Why  or  why  not? 

1 5.  Is  there  anything  else  I  ought  to  know  about  how  the  hospital  implemented  the  outreach 
program?  Did  I  miss  anything? 

Thank  you  for  your  time. 


F-2 


USACHPPM  REPORT  NO.  23-KG-0BS6-09,  Jun-Sep  09 


APPENDIX  G 

TABLE  OF  PROGRAM  ACTIVITIES 


Date 

Unit  Surveys 

Trainings 

Unit  Leader 

Consultation 

Soldier 

Consultation 

Events 

Contacts 

Events 

Contacts 

FEB  07 

4 

200 

0 

8 

5 

473 

1 

23 

6 

3 

APR  07 

1 

29 

3 

196 

7 

23 

MAY  07 

0 

7 

85 

6 

46 

JUN  07 

9 

618 

4 

114 

0 

3 

JUL07 

1 

78 

1 

40 

2 

0 

AUG  07 

3 

153 

2 

89 

2 

0 

SEP  07 

9 

3 

198 

2 

0 

OCT  07 

I 

73 

1 

12 

2 

5 

NOV  07 

I 

32 

4 

337 

0 

32 

DEC  07 

1 

239 

0 

0 

10 

10 

901 

2 

18 

8 

20 

FEB  08 

9 

707 

4 

282 

7 

17 

MAR  08 

8 

1270 

1 

25 

1 

13 

907 

0 

1 

32 

MAY  08 

1 

44 

I 

4 

0 

11 

11 

2 

147 

0 

9 

JUL  08 

7 

416 

1 

13 

1 

2 

AUG  08 

7 

3 

68 

0 

6 

SEP  08 

13 

911 

10 

618 

8 

OCT  08 

1 

41 

3 

404 

0 

2 

NOV  08 

0 

0 

2 

66 

0 

4 

DEC  08 

0 

4 

92 

3 

8 

JAN  09 

0 

4 

192 

0 

7 

FEB  09 

2 

85 

0 

4 

MAR  09 

2 

233 

S 

699 

8 

26 

APR  09 

1 

5 

4 

182 

11 

14 

MAY  09 

2 

24 

8 

895 

5 

34 

JUN  09 

1 

10 

13 

579 

1 

27 

JUL  09 

3 

608 

10 

856 

25 

75 

Total  to  Date 

124 

10174 

108 

6319 

106 

448 

Average/month 

4.13 

339.13 

3.60 

210.63 

3.53 

14.93 
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APPENDIX  H 

SUGGESTED  METRICS  FOR  OUTREACH  PROGRAM  EVALUATIONS 
Process  Evaluation:  How  well  and  to  what  extent  is  the  Outreach  Program  being  implemented? 


Metrics 

How  Measured? 

/  Prevention 

1^;  It !  |  111  pf ;  Sf!  1 1'  j  s ! ;  ti’T*  1 ''  i  >1  nil  1  ll' >  <  ’ ;  s  s " ' '  ml  ill 

Number  of  outreach  services  provided  to  Soldiers  per  year 

Program  records 

r|  Number  of  Soldiers  contacted:  through  each- outreach  service 
'■per  year  '  J'OoMr: j to;: ff Ohittl?!? 

Program  records  ’  |  11  1||| 

i  1  i  Pi 1  ll  1 8|  1  1  >  i  f>  s  if  is  1 1 .  fiji  a  pa  i 

Number  of  each  type  of  referral  resulting  from  outreach 
services  per  year 

Program  records 

■t 

.  •  1 !  1  U  :■  1 

Soldiers’  satisfaction  with  survey 

Satisfaction  survey 

Implementation 

Detailed  description  of  program  activities 

Policy /protocol 

Interviews  with  program 
staff 

Detailed  description  of  program  staffing  levels  and 

nHundii 

Detailed  description  of  how  program  is  incorporated  into  MTF 

SOP/policy 

Interviews  with  program 
staff 

Detailed  description  of  program  funding  and  revenue 

i<  •  ir,  ,  j\  ’  ‘ 

v  ''  '  '  •  '  ■  ,  ■  ■;  ;  •.  ,  «’''<>»>'*  ;.<>  ,  n  t'  '  '  '  « 

SOP.  policy 

AHLTA 
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Intermediate  Outcome  Evaluation:  Has  the  Outreach  Program  met  its  objectives? 


Indicator 

How  Measured? 

Knowledge:  COSC* 

Soldier  can  identify  signs/symptoms  of  combat  stress  reactions 

Survey  before  and  after 
briefings 

d :  Soldier  can  idcnti  1  y  available  heal  th  and  vs  cl  fare  resources 

'  1  ; 'hi pt )  ] ;  p;  -a;  - ) fm i •  >  i  4  d  Hlo ’:$•!$ 'dl ) h; h i 'jMl ' iih v; dj s ; ifi ■' Iv,k 

iilillPiiiPiiiiiSii 

brtefingsli  1 1 IHHMre I 

Soldier  can  describe  self-care  methods 

Survey  before  and  after 
briefings 

Behavior  -Help-seeking 

Number  of  commander  referrals 

Program  data 

Commander  support 

Surveys  with  commanders 

Number  of  Soldier  walk-ins  per  outreach  activity 

Tracking  cards  distributed 
during  outreach 

Additional  intake  question 

Barriers  to  menial  health 

iparrierlsufveydliilllil 

Number  of  Soldiers  who  access  services  of  number  referred 

PDHA/PDHRA 

AHLTA 

Chart  review 

*  Knowledge  pre-/post-survey  can  be  adapted/interchangeable  basec 
(anger,  sleep,  communication,  stress,  etc.) 

Impact  Outcome  Evaluation:  Can  changes  in  the  Soldier  community 
Outreach  Program? 

on  the  content  of  briefings 

be  attributed  to  the 

Indicator 

How  Measured? 

Prevention 

1 HHHHHHHHHI 

Number  of  Soldiers  in  crisis 

OQ-45  scores 

Timeliness  d 

Soldier  referred  to  appropriate  service 

Survey' 

Amount  of  time  between  referral  and  access  to  appropriate  : 
service  d 

PDHA/PDIIRA  Hf Ip 
Mil  1  \ 

s  i  1  >  i  i  ihi  *  b»  M  i' 

Chart  ie\  u 

Long-term 

Soldiei  attrition  rates  of  those  seeking  behavioral  health  services:,i 

Defense  Medical  ll||ilil| 
Surveillance  System 
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